I may be confused, but it seems to me that the issue in generalizing from decision utility to utilitarian utility simply comes down to making an assumption allowing utilities among different people to be compared- to put them on the same scale. I think there’s a pretty strong argument that we can do so, springing from the fact that we all are running essentially the same neural hardware. Whatever experiential value is, it’s made of patterns of neural firing, and we all have basically the same patterns. While we don’t run our brains exactly the same, the mood- and reward-processing circuitry are pretty tightly feedback-controlled, so saying that everyone’s relative utilities are equal shouldn’t be too far from the truth.
But that’s when one adopts an unbiased view. Neither I nor (almost?) anyone else in history have done so. We consider our own happiness more important than anyone else’s. We weight it higher in our own decisions, and that’s perfectly rational. The end point of this line of logic is that there is no objective ethics—it’s up to the individual.
But there is one that makes more sense than others when making group decisions, and that’s sum utilitarianism. That’s the best candidate for an AI’s utility function. Approximations must be made, but they’re going to be approximately right. They can be improved by simply asking people about their preferences.
The common philosophical concern that you can’t put different individuals preferences on the same scale does not hold water when held up against our current knowledge about how brains register value and so create preferences.
I may be confused, but it seems to me that the issue in generalizing from decision utility to utilitarian utility simply comes down to making an assumption allowing utilities among different people to be compared- to put them on the same scale. I think there’s a pretty strong argument that we can do so, springing from the fact that we all are running essentially the same neural hardware. Whatever experiential value is, it’s made of patterns of neural firing, and we all have basically the same patterns. While we don’t run our brains exactly the same, the mood- and reward-processing circuitry are pretty tightly feedback-controlled, so saying that everyone’s relative utilities are equal shouldn’t be too far from the truth.
But that’s when one adopts an unbiased view. Neither I nor (almost?) anyone else in history have done so. We consider our own happiness more important than anyone else’s. We weight it higher in our own decisions, and that’s perfectly rational. The end point of this line of logic is that there is no objective ethics—it’s up to the individual.
But there is one that makes more sense than others when making group decisions, and that’s sum utilitarianism. That’s the best candidate for an AI’s utility function. Approximations must be made, but they’re going to be approximately right. They can be improved by simply asking people about their preferences.
The common philosophical concern that you can’t put different individuals preferences on the same scale does not hold water when held up against our current knowledge about how brains register value and so create preferences.
That’s a big leap. Why would weighing the quality of our own experiences more highly mean that there’s no objective ethics?